Search for: All records

Identifying and annotating student use of debugging strategies when solving computer programming problems can be a meaningful tool for studying and better understanding the development of debugging skills, which may lead to the design of effective pedagogical interventions. However, this process can be challenging when dealing with large datasets, especially when the strategies of interest are rare but important. This difficulty lies not only in the scale of the dataset but also in operationalizing these rare phenomena within the data. Operationalization requires annotators to first define how these rare phenomena manifest in the data and then obtain a sufficient number of positive examples to validate that this definition is reliable by accurately measuring Inter-Rater Reliability (IRR). This paper presents a method that leverages Large Language Models (LLMs) to efficiently exclude computer programming episodes that are unlikely to exhibit a specific debugging strategy. By using LLMs to filter out irrelevant programming episodes, this method focuses human annotation efforts on the most pertinent parts of the dataset, enabling experts to operationalize the coding scheme and reach IRR more efficiently. 

Extensive research underscores the importance of stimulating students' interest in learning, as it can improve key educational outcomes such as self-regulation, collaboration, problem-solving, and overall enjoyment. Yet, the mechanisms through which interest manifests and impacts learning remain less explored, particularly in open-ended game-based learning environments like Minecraft. The unstructured nature of gameplay data in such settings poses analytical challenges. This study employed advanced data mining techniques, including changepoint detection and clustering, to extract meaningful patterns from students' movement data. Changepoint detection allows us to pinpoint significant shifts in behavior and segment unstructured gameplay data into distinct phases characterized by unique movement patterns. This research goes beyond traditional session-level analysis, offering a dynamic view of the learning process as it captures changes in student behaviors while they navigate challenges and interact with the environment. Three distinct exploration patterns emerged: surface-level exploration, in-depth exploration, and dynamic exploration. Notably, we found a negative correlation between surface-level exploration and interest development, whereas dynamic exploration positively correlated with interest development, regardless of initial interest levels. In addition to providing insights into how interest can manifest in Minecraft gameplay behavior, this paper makes significant methodological contributions by showcasing innovative approaches for extracting meaningful patterns from unstructured behavioral data within game-based learning environments. The implications of our research extend beyond Minecraft, offering valuable insights into the applications of changepoint detection in educational research to investigate student behavior in open-ended and complex learning settings. 

Debugging is a challenging task for novice programmers in computer science courses and calls for specific investigation and support. Although the debugging process has been explored with qualitative methods and log data analyses, the detailed code changes that describe the evolution of debugging behaviors as students gain more experience remain relatively unexplored. In this study, we elicited “constituents” of the debugging process based on experts’ interpretation of students’ debugging behaviors in an introductory computer science (CS1) course. Epistemic Network Analysis (ENA) was used to study episodes where students fixed syntax/checkstyle errors or test errors. We compared epistemic networks between students with different prior programming experience and investigated how the networks evolved as students gained more experience throughout the semester. The ENA revealed that novices and experienced students put different emphasis on fixing checkstyle or syntax errors and highlighted interesting constituent co-occurrences that we investigated through further descriptive and statistical analyses. 

Debugging is a challenging task for novice programmers in computer science courses and calls for specific investigation and support. Although the debugging process has been explored with qualitative methods and log data analyses, the detailed code changes that describe the evolution of debugging behaviors as students gain more experience remain relatively unexplored. In this study, we elicited “constituents” of the debugging process based on experts’ interpretation of students’ debugging behaviors in an introductory computer science (CS1) course. Epistemic Network Analysis (ENA) was used to study episodes where students fixed syntax/checkstyle errors or test errors. We compared epistemic networks between students with different prior programming experience and investigated how the networks evolved as students gained more experience throughout the semester. The ENA revealed that novices and experienced students put different emphasis on fixing checkstyle or syntax errors and highlighted interesting constituent co-occurrences that we investigated through further descriptive and statistical analyses. 

Abstract BackgroundLearning analytics (LA) research often aggregates learning process data to extract measurements indicating constructs of interest. However, the warranty that such aggregation will produce reliable measurements has not been explicitly examined. The reliability evidence of aggregate measurements has rarely been reported, leaving an implicit assumption that such measurements are free of errors. ObjectivesThis study addresses these gaps by investigating the psychometric pros and cons of aggregate measurements. MethodsThis study proposes a framework for aggregating process data, which includes the conditions where aggregation is appropriate, and a guideline for selecting the proper reliability evidence and the computing procedure. We support and demonstrate the framework by analysing undergraduates' academic procrastination and programming proficiency in an introductory computer science course. Results and ConclusionAggregation over a period is acceptable and may improve measurement reliability only if the construct of interest is stable during the period. Otherwise, aggregation may mask meaningful changes in behaviours and should be avoided. While selecting the type of reliability evidence, a critical question is whether process data can be regarded as repeated measurements. Another question is whether the lengths of processes are unequal and individual events are unreliable. If the answer to the second question is no, segmenting each process into a fixed number of bins assists in computing the reliability coefficient. Major TakeawaysThe proposed framework can be a general guideline for aggregating process data in LA research. Researchers should check and report the reliability evidence for aggregate measurements before the ensuing interpretation.